Multiresolution Instance-Based Learning

Instance-based learning methods explicitly remem­ ber all the data that they receive They usually have no training phase and only at prediction time do they perform computation Then they take a query search the database for similar datapoints and build an on-line local model (such as a local average or local regression) with which to predict an output value In this paper we review the advantages of instance based methods for autonomous systems but we also note the ensuing cost hopelessly slow computation as the database grows large We present and evaluate a new way of structuring a database and a new algorithm for accessing it that maintains the advantages ot instance-based learn ing Earlier attempts to combat the cost of instancebased learning have sacrificed the explicit retention of all data or been applicable only to instancebased predictions based on a small number of near neighbors, or have had to re-,ntrodtice an exp}tctt training phase in the form ot an interpolative data structure Our approach builds a multiresolution data structure to summarize the database of experi­ ences at all resolutions of interest simultaneously This permits us to query the database with the same flexibility as a conventional linear search but at greatly reduced computational cost 1 I n t r o d u c t i o n Instance-based learning methods [Stanhll et al 1986 Atkeson, 1989 Ahaet al 1991 Moore 1990] are highly flexible general purpose techniques tor making predic­ tions from earlier data Instance based methods (also known as memory-based methods or lazy-learning methods and closely related to case-based' methods) explicitly remember all the data they are shown Only at prediction time do they perform non-tnvial amounts of computation This behavior differs from more conven­ tional machine learning algorithms in which training occurs between the reception of data and prediction Examples of instance based methods are nearest neighbor kernel regression and locally weighted linear regression Example of non-instance-based techniques (they have a training phase) are neural networks and decision trees Instance based methods can sometimes be a preferable form of function approximator There are three main rea sons for this Flexible Inductive Bias With very little data a method such as nearest neighbor gives sensible conservative predictions it does not wildlv extrapolate But as the amount of data increases so does the complexity of the function that nearest neighbor can approximate This contrasts with for example multi-layer neural networks that do not by default have this property of representative power increasing locally according to the amount of local data In the limit very local methods can learn any piecewise continuous function to arbitrary preci sion (although with high dimensional uniformly distrib­ uted input the amount of data to do this can be enormous) For practical use in function approximation, much better instance-based methods than nearest neighbor are avail­ able that form local linear models and compute weighted averages of data to remove the noise from predictions (e g see(Atkeson 1989 Grosse I989]) Learning parameters need not be fixed in advanc e There are many learning parameters in instance-based algorithms One ot the most important concerns the extent 10 which the smoothing of noise is traded against goodness of fit Others include (1) the parameters of a distance metric for determining the similarity between an input point and the query and (II) the discrete decision of which attributes are relevant Instance based methods do not need to decide on these learning parameters in advance They can use whichever parameters they desire tor one prediction and then have the option of using an entirely different sel for another prediction This is of immense use in an autono mous system that is both making new predictions online and tuning its learning parameters online as new data is arriving |Moore el al 1992] In contrast a non instancebased method must choose a parameter set and then train with it If a different parameter set is later needed then it is necessary for a non-instance-based method to retrain itself (and it is therefore also necessary for it to remember all previous data) Instance based can cover the global local spectrum Instance based methods do not necessarily have to be local predictors based on a small handful of local datapoints This is particularly important lor large num­ bers of attributes, highly noisy data and for small data bases Many of our own applications involve very noisy systems in which the underlying function is non linear but generally smooth In these cases the best instance-based function approximator might for example use the closest 30% of all datapoints to the query to form its prediction In